Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: adding multiprocessing support #441

Merged
merged 3 commits into from
Apr 12, 2021
Merged

feature: adding multiprocessing support #441

merged 3 commits into from
Apr 12, 2021

Conversation

domanchi
Copy link
Contributor

@domanchi domanchi commented Apr 1, 2021

I never knew parallel processing could be so easy. ❤️ python

Summary

This increases the efficiency of scans by throwing some parallel processing to it.

Since scanning files are independent of each other, it seemed like a prime candidate for this. I considered parallelizing line scanning, but then realized that this may not be such a good idea, since lines are somewhat dependent on each other (e.g. context-based filters).

Testing Done

I ran this on our internal monolith, and these were the stats I got:

# Some stats on the repository itself
$ git ls-files | wc -l
35519
$ cloc $(git ls-files)
...
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
...
--------------------------------------------------------------------------------
SUM:                          22948         660907         422368        3680522
--------------------------------------------------------------------------------
$ lscpu
...
CPU(s):              16

# Running with single process
$ time detect-secrets scan > /tmp/single.baseline
real    23m41.917s
user    23m20.518s
sys     0m3.527s

# Running with multiprocessor
$ time detect-secrets scan > /tmp/mp.baseline
real    6m4.887s
user    37m4.611s
sys     0m8.304s

# Comparing results
$ diff /tmp/single.baseline /tmp/mp.baseline
4961c4961
<   "generated_at": "2021-04-01T14:42:50Z"
---
>   "generated_at": "2021-04-01T15:05:30Z"

That's like a 74% optimization. Heck yeah!

calvinli
calvinli approved these changes Apr 12, 2021
View reviewed changes
Copy link
Member

@calvinli calvinli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a quick peek and it does look like scan.scan_file is independent/doesn't rely on any shared state so this should work.

detect_secrets/core/secrets_collection.py Show resolved Hide resolved
@domanchi domanchi merged commit e220759 into master Apr 12, 2021
killuazhu pushed a commit to IBM/detect-secrets that referenced this pull request Jun 29, 2021
@justineyster
Bump to version 0.13.1+ibm.39.dss to fix vulnerabilities (Yelp#441)
c279f78 
Co-authored-by: detect-secrets-updater <detect-secrets-updater@ibm.com>
Co-authored-by: JUSTIN EYSTER <Justin.Eyster@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@calvinli calvinli calvinli approved these changes

Assignees

@calvinli calvinli

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@domanchi @calvinli